Image data delivery system, image data transmitting device thereof, and image data receiving device thereof

ABSTRACT

A server ( 1 ) analyses the request information transmitted from a client ( 11 ) by means of a request analyzer  4  and selects the necessary image data from multiple viewpoint image data ( 2 ) to output it to an image generator ( 3 ), where image data for the requested viewpoint is generated by interpolation to be output to an image synthesizer ( 5 ). In the image synthesizer ( 5 ), a plurality of images data are synthesized in a form suitable for encoding. An encoder ( 6 ) encodes the image data at a suitable bit rate and transfers it to a network ( 7 ). The client ( 11 ) receives the encoded image data, decodes the data through a decoder ( 12 ) and outputs the decoded image data to an image processor ( 13 ), where the image data is converted into an appropriate form in conformity with a stereoscopic display format and is displayed on a display unit ( 14 ). The client ( 11 ) also includes an input unit ( 16 ) for change of the viewpoint, and transmits the request information of viewpoint alternation to the network ( 7 ) by means of a request output unit ( 15 ). Thus this system enables observation of a stereoscopic image viewed from an arbitrary viewpoint even with a mobile terminal or the like.

TECHNICAL FIELD

The present invention relates to an image data distribution system fordistributing image data and an image data transmitting apparatus andimage data receiving apparatus for the system.

BACKGROUND ART

Conventionally, there have been proposed devices for displaying astereoscopic image viewed from an arbitrary viewpoint using multipleviewpoint images taken by a plurality of cameras. One example of such adisplay device is shown in FIG. 16. This device includes: an input unit95 for setting the left and right eye viewpoints; a computer 91connected to input unit 95 to generate image data that constitutes twoimages viewed from the left and right eye viewpoints; and a stereoscopicdisplay 98 that is connected to computer 91 to receive image data of thetwo images and makes stereoscopic display by synthesizing the datathrough an interior circuit thereof.

Computer 91 comprises: a memory 92 for storing multiple viewpoint imagedata; a CPU 93 for implementing a process of generating image data thatconstitutes two images; an input/output interface 94 connected tostereoscopic display 98 and input unit 95 for controlling the inputvalues from input unit 95 and the input and output of image data; and abus 96 for mutually connecting CPU 93, memory 92 and input/outputinterface 94.

From the multiple viewpoint image data stored in memory 92, one desiredset of image data is selected while its left and right eye viewpointsfor implementing stereoscopic display are designated by using the inputunit 95. In this case, it is judged whether the image data viewed fromthe left eye viewpoint is present in the multiple-view data. If the datais present, the image data is output through input/output interface 94.If the data is not present, the image data that is viewed from the lefteye viewpoint is generated by an interpolating process using the imagedata constituting a predetermined plurality of images taken from theviewpoints adjacent to the left eye viewpoint, and the generated data isoutput through input/output interface 94. A similar process is carriedout from the right eye viewpoint.

However, since the conventional technology needs high enough memory anda high-performance CPU, it suffers from the problem of being impossiblefor a terminal such as a mobile terminal, which has only limited memoryand a CPU with a limited processing performance, to realize thetechnology.

Further, since it is assumed in the conventional technology that aspecial stereoscopic display is directly connected to the computer, noconsideration is taken for efficient transmission of image data, so thatit is impossible to deal with the case where various kinds ofstereoscopic displays are connected through a network.

The present invention has been devised under the above circumstances, itis therefore an object of the present invention to provide an image datadistribution system, its image data transmitting apparatus and imagedata receiving apparatus, which enable observation of a stereoscopicimage viewed from a viewpoint even with a mobile terminal or the like.

Another object is to enable identical original image data to be used forobservations of a stereoscopic image viewed from arbitrary viewpointsthrough various kinds of stereoscopic displays connected by a networkwhile realizing efficient transmission of stereoscopic image data.

DISCLOSURE OF INVENTION

In order to achieve the above objects, the image data distributionsystem according to the present invention and its image datatransmitting apparatus and image data receiving apparatus employ thefollowing means.

Specifically, an image data distribution system of the present inventionincludes: a request information receiving means for receiving client'srequest information transmitted by way of a network; a requestinformation analyzing means for analyzing the request informationreceived by the request information receiving means; a multipleviewpoint image supply means which selects necessary image data fromcoded and stored multiple viewpoint image data of images taken by aplurality of cameras, based on viewpoint information from the requestinformation analyzed by the request information analyzing means anddecodes and supplies selected data; an image generating means which,based on image data supplied from the multiple viewpoint image supplyingmeans, generates image data of an image viewed from a predeterminedviewpoint in conformity with the request information; an imagesynthesizing means for synthesizing a plurality of images data generatedby the image generating means, based on display unit information fromthe request information; a coding means for coding image datasynthesized by the image synthesizing means; a transmitting means fortransmitting coded image data by the coding means to the network; areceiving means for receiving the coded image data via the network; adecoding means for decoding the coded image data received by thereceiving means; an image processing means for processing decoded imagedata by the decoding means so as to be display able on a display means;the display means for displaying image data processed by the imageprocessing means; a request information input means for allowing inputof the client's request information; and a request informationtransmitting means for transmitting the request information to thenetwork.

Also, an image data transmitting apparatus of the present inventionincludes: a request information receiving means for receiving client'srequest information transmitted by way of a network; a requestinformation analyzing means for analyzing the request informationreceived by the request information receiving means; a multipleviewpoint image supply means which selects necessary image data fromcoded and stored multiple viewpoint image data of images taken by aplurality of cameras, based on viewpoint information from the requestinformation analyzed by the request information analyzing means anddecodes and supplies selected data; an image generating means which,based on image data supplied from the multiple viewpoint image supplyingmeans, generates image data of an image viewed from a predeterminedviewpoint in conformity with the request information; an imagesynthesizing means for synthesizing a plurality of images data generatedby the image generating means, based on display unit information fromthe request information; a coding means for coding image datasynthesized by the image synthesizing means; and a transmitting meansfor transmitting coded image data by the coding means to the network.

Further, the image data distribution system of the present invention mayfurther include a management information adding means for addingmanagement information for enabling access to the image data ofindividual viewpoints and random access, to the multiple viewpoint imagedata.

Moreover, the image data transmitting apparatus of the present inventionmay further include a management information adding means for addingmanagement information for enabling access to the image data ofindividual viewpoints and random access, to the multiple viewpoint imagedata.

Still, an image data distribution system of the present inventionincludes: a request information receiving means for receiving client'srequest information transmitted by way of a network; a requestinformation analyzing means for analyzing the request informationreceived by the request information receiving means; a multipleviewpoint image supply means for supplying multiple viewpoint imagedata; an image generating means which, based on viewpoint informationfrom the request information analyzed by the request informationanalyzing means, receives input of necessary image data from themultiple viewpoint image supply means and generates image data of animage viewed from a predetermined viewpoint in conformity with therequest information; an image synthesizing means for synthesizing aplurality of images data generated by the image generating means, basedon display unit information from the request information; a coding meansfor coding image data synthesized by the image synthesizing means; atransmitting means for transmitting coded image data by the coding meansto the network; a receiving means for receiving the coded image data viathe network; a decoding means for decoding the coded image data receivedby the receiving means; an image processing means for processing decodedimage data by the decoding means so as to be display able on a displaymeans; the display means for displaying image data processed by theimage processing means; a request information input means for allowinginput of the client's request information; a request informationtransmitting means for transmitting the request information to thenetwork; and a judgement means for judging whether the received imagedata is of two-dimensional image data or stereoscopic image data.

Another image data receiving apparatus of the present inventionincludes: a receiving means for receiving coded image data by way of anetwork; a decoding means for decoding the coded image data received bythe receiving means; an image processing means for processing decodedimage data by the decoding means so as to be display able on a displaymeans; the display means for displaying image data processed by theimage processing means; a request information input means for allowinginput of request information of a client; a request informationtransmitting means for transmitting the request information to thenetwork; and a judgement means for judging whether the received imagedata is of two-dimensional image data or stereoscopic image data.

Next, an image data distribution system of the present inventionincludes: a request information receiving means for receiving client'srequest information transmitted by way of a network; a requestinformation analyzing means for analyzing the request informationreceived by the request information receiving means; a multipleviewpoint image supply means for supplying multiple viewpoint imagedata; an image generating means which, based on viewpoint informationfrom the request information analyzed by the request informationanalyzing means, receives input of necessary image data from themultiple viewpoint image supply means and generates image data of animage viewed from a predetermined viewpoint in conformity with therequest information; an image synthesizing means for synthesizing aplurality of images data generated by the image generating means, basedon display unit information from the request information; a coding meansfor coding image data synthesized by the image synthesizing means; atransmitting means for transmitting coded image data by the coding meansto the network; a receiving means for receiving the coded image data viathe network; a decoding means for decoding the coded image data receivedby the receiving means; an image processing means for processing decodedimage data by the decoding means so as to be display able on a displaymeans; the display means for displaying image data processed by theimage processing means; a request information input means for allowinginput of the client's request information; a request informationtransmitting means for transmitting the request information to thenetwork; and an identification information adding means for adding tothe image data to be transmitted a piece of information that indicateswhether the image data is of two-dimensional image data or stereoscopicimage data.

Further, an image data transmitting apparatus of the present inventionincludes: a request information receiving means for receiving client'srequest information transmitted by way of a network; a requestinformation analyzing means for analyzing the request informationreceived by the request information receiving means; a multipleviewpoint image supply means for supplying multiple viewpoint imagedata; an image generating means which, based on viewpoint informationfrom the request information analyzed by the request informationanalyzing means, receives input of necessary image data from themultiple viewpoint image supply means and generates image data of animage viewed from a predetermined viewpoint in conformity with therequest information; an image synthesizing means for synthesizing aplurality of images data generated by the image generating means, basedon display unit information from the request information; a coding meansfor coding image data synthesized by the image synthesizing means; atransmitting means for transmitting coded image data by the coding meansto the network; and an identification information adding means foradding to the image data to be transmitted a piece of information thatindicates whether the image data is of two-dimensional image data orstereoscopic image data.

Moreover, an image data receiving apparatus of the present inventionincludes: a receiving means for receiving coded image data by way of anetwork; a decoding means for decoding the coded image data received bythe receiving means; an image processing means for processing decodedimage data by the decoding means so as to be display able on a displaymeans; the display means for displaying image data processed by theimage processing means; a request information input means for allowinginput of request information of a client; a request informationtransmitting means for transmitting the request information to thenetwork; and an identification information adding means for adding tothe received image data a piece of information that indicates whetherthe image data is of two-dimensional image data or stereoscopic imagedata.

As has been described heretofore, since the above configuration makes itunnecessary to provide large amounts of memory and a high-performanceCPU on the client side, the invention has the effect of enablingobservation of a stereoscopic image viewed from an arbitrary viewpointeven with a mobile terminal and the like.

The present invention also has the effect of enabling use of anidentical original image data to provide observations of a stereoscopicimage viewed from arbitrary viewpoints through various types ofstereoscopic displays that are connected by way of a network.

BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a block diagram showing anembodiment of an image data distribution system according to the presentinvention.

FIG. 2 is an illustration showing layouts of a plurality of cameras forpreparing multiple-view image data.

FIG. 3 is an illustration showing the left eye viewpoint Land the righteye viewpoint R for which image data is generated by interpolation.

FIG. 4 is an illustration showing areas to be cut out from the generatedimage data, depending on the resolution of a display unit.

FIG. 5 is an illustrative view showing a mobile terminal to be a client.

FIG. 6 is an illustration showing examples of states of stored data ofmultiple-view image data.

FIG. 7 is an illustration showing examples of states of generated imagedata.

FIG. 8 is a diagram showing storage and extraction of multiple-viewimage data.

FIG. 9 is a flowchart showing the sequence of processing on a server.

FIG. 10 is a flowchart showing the sequence of processing on a client.

FIG. 11 is an illustration showing an example of coding multipleviewpoint moving picture data based on MPEG-4.

FIG. 12 is a diagram showing multiple viewpoint image data added withmanagement information.

FIG. 13 is a chart showing one example of management information.

FIG. 14 shows an expected connection state between a server and clientsin the present embodiment.

FIG. 15 is a flowchart showing details of the processing of an imagegenerator.

FIG. 16 is a diagram showing a conventional example.

BEST MODE FOR CARRYING OUT THE INVENTION

The embodiment of an image data distribution system according to thepresent invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing the embodiment of an image datadistribution system according to the present invention.

A server 1 (image data transmitting apparatus) and a client 11 (imagedata receiving apparatus) are connected to each other by way of anetwork 7. Server 1 has stored (recorded) multiple viewpoint image data2. When client 11 transmits request information to server 1, the clientis able to implement stereoscopic display of an image viewed from adesired viewpoint on a display unit 14.

Here, the multiple viewpoint image data has not been necessarily stored(recorded) previously in server 1, but may be supplied in real time fromwithout.

Server 1 analyses the request information transmitted from client 11 bya request analyzer 4 (request information analyzing means (includingrequest information receiving means)) and selects the necessary imagedata from multiple viewpoint image data 2 (multiple viewpoint imagesupply means) to output it to an image generator 3 (image generatingmeans) where image data for the requested viewpoint (viewpointinformation) is generated by interpolation to be output to an imagesynthesizer 5 (image synthesizing means). In image synthesizer 5, aplurality of supplied images data are synthesized in a form (formatbased on the display unit information) suitable for encoding to beoutput to an encoder 6 (coding means). In encoder 6, the supplied imagedata is encoded at a suitable bit rate to be transmitted to network 7(transmitting means).

Client 11 receives the coded image data (receiving means), and decodesthe data through a decoder 12 (decoding means) and outputs the decodedimage data to an image processor 13 (image processing means), where theimage data is converted into an appropriate form in conformity with astereoscopic display format so that the image data is displayed on adisplay unit 14 (display means). Client 11 also includes an input unit16 (request information input means) for change of the viewpoint, andtransmits the request information of viewpoint alternation to network 7by way of a request output unit 15 (request information transmittingmeans).

Now, server 1 and client 11 will be described in detail.

Multiple viewpoint image data 2 is formed of a set of images data takenby a plurality of cameras. The plurality of cameras are typically laidout as shown in FIG. 2(a) so that the optical axes of the plurality ofcameras intersect at one point. As a special example, the cameras maybearranged on the circumference of a circle so that the optical axes ofthe cameras are directed to the center of the circle, as shown in FIG.2(b). In either case, the cameras are not necessarily arrangedequi-distantly, but may be laid out in some parts densely and otherssparsely. The information as to how the cameras are allocated is alsorecorded together with the image data. This allocation information isused to determine the image data of which cameras should be used whenimage generator 3 generates the image data from a designated viewpointby interpolation based on the request information from client 11.

Next, description will be made about the necessary camera image datawhen the image of the requested viewpoint is generated by interpolation.In the example shown in FIG. 3(a), the left eye viewpoint L and righteye viewpoint R are designated as illustrated with respect to cameras C1to C4. In this case, the images of data from C1 and C2 are used togenerate the image data for left eye viewpoint Land the images of datafrom C2 and C3 are used to generate the image data for right eyeviewpoint R. Similarly, in the example shown in FIG. 3(b), the layout ofcameras C1 to C4 and the requested left eye viewpoint L and right eyeviewpoint R are positioned as illustrated. In this case, both the imagedata for left eye viewpoint L and the image data for right eye viewpointR are generated based on the images of data from C1 and C2. Though fourcameras are used in the examples in FIG. 3, the number of cameras is notlimited to four.

Generation of images from an intermediate viewpoint by interpolation isa known technology, and is disclosed in detail in, for example,“Tsunashima et al.: generation of intermediate image data from atwo-view stereoscopic image data taking occlusion into consideration, 3DImage Conference '95, pp. 174-177 (1995)”, and “Azuma et al.: parallaxestimation based on the edge information for generation of intermediateimage data, 3D Image Conference '95, pp. 190-194 (1995)”, and others.

When the necessary viewpoint image data has been generated at imagegenerator 3, image synthesizer 5 implements an extraction process ofimage data in an amount for the requested resolution. In FIG. 4(a), thesize of the generated image data 21 is assumed to be equal to the sizeof the image taken by the camera, and the resolution required fordisplay on client 11 is shown by an area 22. In FIG. 4(a), only part ofthe generated image data is cut out. Alternatively, as shown in FIG.4(b), instead of cutting out an image in the resolution required fordisplay on client 11, the maximum area 24 capable of being cut out whilekeeping the display aspect may be cut out from the generated image data23 (which is assumed to be the same size as the image taken by thecamera), then may be reduced to the required resolution. Also, imagegenerator 3 may be adapted to generate only the necessary amount ofimage data for the required resolution.

FIG. 5 is an illustrative view showing a mobile terminal 50 to be aclient 11. The terminal includes a stereoscopic display 54, a four-wayarrow key 51 for moving the viewpoint up, down, left and right and keys52 and 53 for moving the viewpoint forward and backward. Thoughunillustrated, the terminal also has a communication means for makingcommunication with the server.

FIG. 6 shows examples of storage states of multiple viewpoint image data2. Because moving picture data taken by a plurality of cameras need tobe temporally synchronized to each other, the plurality of images dataC1 to C4 taken by different cameras may be stored in such a manner as tojoin them abreast and form a piece of image data made of one image, asshown in FIG. 6(a). This format assures that the images data C1 to C4contained in the image data of the single image were taken at the sametime, leading to easy time management. The way of joining is not limitedto a line abreast as shown in FIG. 6(a) but the images may be joined asshown in FIG. 6(b), for example.

In contrast, as shown in FIG. 6(c), a format of separately storing theimages of data C1 to C4 taken by different cameras may be alsoconsidered. The advantage of this method is that, when camera images C1and C2 alone are needed as shown in FIG. 3(b) in order to generate theimage data from the requested viewpoint, these can be easily picked up.

Multiple viewpoint image data 2 may be stored either by being compressedor non-compressed. Here, referring to FIG. 8, description will be madeabout a case where individual pieces of camera image data which aregiven separately in the manner as shown in FIG. 6(c) are stored afterbeing data compressed. In this case, images of data C1 to C4 taken bythe individual cameras are input to an encoder 31 as shown in FIG. 8(a).Encoder 31 encodes each image data and outputs the necessary information(frame type, generation bit count, etc.) for generating managementinformation, to a management information generator 32. In a recorder 33,the coded image data and the management information are recorded as thestorage data (management information adding means). The detail of themanagement information and storage format will be described later.

When the storage data which has been recorded in the manner shown inFIG. 8(a) is used as the multiple viewpoint image data, the data needsto be decoded so as to be handled by image generator 3. In this case, asshown in FIG. 8(b), a selector 34, in accordance with the request fromthe client, selects only the necessary image data from the stored dataand outputs to a decoder 35 where the data is decoded, whereby it ispossible to obtain the necessary and sufficient original image data (theimage data to be used by image generator 3).

In this process, in order to extract the necessary part quickly, themanagement information recorded together with image data is utilized.

FIG. 7 shows examples of image data states generated by imagesynthesizer 5. Though the left eye viewpoint image data L and the righteye viewpoint image data R can be encoded as separate pieces of imagedata as shown in FIG. 7(b), it is preferred that both images of data arejoined side by side (or up and down) into a piece of synthesized imagedata formed of a single image then the joined image data is encoded.Formation of the image data composed of a single image as shown in FIG.7(a) is able to assure the synchronism between the left eye viewpointimage data L and the right eye viewpoint image data R, leading to easytime management. Particularly, when a coding scheme entailing frameskipping such as MPEG-4 is used, it is possible to prevent occurrence ofsuch a situation where, at a certain moment, the frame of the left eyeviewpoint image data L is present whereas the frame of the right eyeviewpoint image data R is not present. Further, since the image data israte-controlled as a single image, both the left eye viewpoint image Land the right viewpoint image R can be kept substantially equal in imagequality. When the images are encoded as separate pieces of image data,depending on the rate control result there occurs a case where onextraction of a frame at a certain moment the image quality of the lefteye viewpoint image L is good while the image quality of the right eyeviewpoint image R is bad. In such a case, the resultant stereoscopicdisplay presents poor quality. Thus, it is possible to improve thequality of stereoscopic display when the data is adapted to take theform as shown in FIG. 7(a).

Depending on the stereoscopic display format of a client 11, there aresome cases where the image data to be finally displayed on the displayunit 14 takes a form of strips of left eye viewpoint image data Landstrips of right eye viewpoint image data R being alternated with eachother every line as shown in FIG. 7(c) (for lenticular mode, parallaxbarrier mode and the like). In such a case, however, it is preferredthat the image data to be coded takes the form as shown in FIG. 7(a).The reason is that if coding is performed on a block basis as in DCT,the image data having the form as shown in FIG. 7(c) will present weakcorrelation between adjacent pixels, hence high spatial frequencies,producing poor compression efficiency. The same method can be appliedfor the cases where the number of viewpoints is greater than 2.

When images of data for a plurality of viewpoints are joined to form apiece of image data made of a single image as shown in FIG. 7(a), it isimpossible to know the difference between normal two-dimensional imagedata and stereoscopic image data from a format point of view. No problemwill occur when real-time streaming is handled using the system shown inFIG. 1 because image data is transmitted in real time in accordance withthe request from the client side. However, in a case where the thustransmitted image data has been once recorded locally on the client 11side and is played afterward, it is impossible to distinguish whether itis of two-dimensional image data or stereoscopic image data. In order toprevent this, when any piece of image data having the form as shown inFIG. 7(a) is recorded, a flag for identification indicating whether theimage data is of two-dimensional image data or stereoscopic image datamay and should be added (identification information adding means). Thisaddition of the identification flag may be done either on server 1 orclient 11. Further, client 11 has a judgement means for distinctionbetween two-dimensional image data or stereoscopic image data.

In connection with the above, when client 11 records image data locally,the recording means should be placed before decoding (upstream ofdecoder 12) or after image processing (downstream of the imageprocessor).

FIG. 9 is a flowchart showing the sequence of processing on server 1.First, the request from a client is analyzed (Step S1). Then, thenecessary image data is selected from multiple viewpoint image data(Step S2). Next, an image from the requested viewpoint is generatedbased on the selected data (Step S3). Subsequently, the image data of asize required for display is cut out (and reduced as required) (StepS4). Then, the cutout left eye viewpoint image data and the cutout righteye viewpoint image data are joined (Step S5). Next, the joined imagedata is encoded (Step S6). Finally, the data is output as a bit stream(Step S7).

FIG. 10 is a flowchart showing the sequence of processing on client 11.First, initialization is done so that the viewpoint position (viewpointinformation) at the initial state and information not dependent on theviewpoint (stereoscopic display format, resolution etc. (display unitinformation)) are set up (Step S11). Next, these pieces of informationare transmitted as a request to server 1 (Step S12). Then, the bitstream (image data) meeting the request is transmitted from the serverby way of the network (Step S13). Subsequently, the bit stream isdecoded (Step S14). Since, as shown in FIG. 7(a) the decoded image datais not in the form that is directly and stereoscopically display able,the data is rearranged so as to conform with the stereoscopic displayformat, as shown in FIG. 7(c) (Step S15). Then the data is displayed ondisplay unit 14 (Step S16). Next, it is judged whether there is a nextdisplay (Step S17). If display is continued, it is judged whether thereis a request for change of the viewpoint (Step S18). Subsequently, ifthe viewpoint should be changed, the request is sent again to server 1and the operation returns to Step S12. When no more display is needed atStep S18, the operation goes to Step S13.

FIG. 14 shows an expected connection state between a server and clientsin this embodiment. Terminals A to C are the clients and are connectedto a server 41 via a network 7. Each terminal has a stereoscopic displayformat and a display resolution different from others, and the positionof the viewpoint for observation is different from others. Therefore,server 41, in accordance with the request from each terminal, transmitsto the terminal different image data from the others. If the wholemultiple viewpoint image data stored (recorded) on server 41 istransmitted to the network so as to permit each terminal to choose thenecessary data to display, transmission can be done with one kind ofimage data. However, since the multiple viewpoint image data is composedof a massive amount of data and because of bandwidth limitations of thenetwork, such a transmission is unfeasible. This is why the transmissionsystem of the present invention in which the server can transmitappropriate image data in accordance with the request from a client isrequisite to an environment in which different kinds of terminals areconnected to a network.

FIG. 15 is a flowchart showing details of the processing in the imagegenerator 3. First, the position of the viewpoint requested from aclient is analyzed (Step S21). Next, it is judged whether there is anyimage whose viewpoint corresponds to the requested viewpoint, among thestored multiple viewpoint image data (Step S22). If there is, the imagedata is directly used; otherwise the interpolation image is generated(Step S23). Subsequently, the data is output as the image data (StepS24).

Next, management information will be described.

FIG. 11 shows one example of coding a multiple viewpoint moving picturebased on MPEG-4. Since frame skipping can be used with MPEG-4, theframes to be coded have discontinuous frame numbers such as LTf0, LTf3,LTf5 and LTf10. Further, in this example, LTf0 is assumed to be coded asan intra frame coding frame (I-frame), LTf10 is assumed to be coded as aframe (P-frame) which is predictive coded based on the decoded frame ofLTf0, and LTf3 and LTf5 are assumed to be coded as frames (B-frames)which are bi-directionally predictive coded based on the decoded framesof LTf0 and LTf10.

FIG. 13 is a chart showing one example of management information addedat management information generator 32 shown in FIG. 8(a). The codeddata of individual camera images can be joined to management informationas shown in FIG. 12 and stored. This management information is theinformation that allows for access to each camera image data. In thecase of a multiple viewpoint moving picture, the management informationcontains not only the information which allows for access to each cameraimage but also the information which allows random access to the codeddata at a designated time within each camera image data.

FIG. 13(a) shows one example of management information for access to thecoded data of individual camera images. For example, it is indicatedthat the coded data of camera image C2 exists at the B2-th byte from thefront of the data in FIG. 12. FIG. 13(a) further includes the pointersto the information for making access to the coded data at designatedtimes within the camera image data. For the coded data of C2, it isindicated that the access table for making access to the coded data atdesignated times is located at address P2 within the managementinformation.

FIG. 13(b) shows one example of an access table to coded data atdesignated times. Times t1, t2, t3, . . . may be set up at regularintervals or may be set up arbitrary intervals apart. For example, it isindicated that the coded data corresponding to time t3 exists at theBt3-th byte from the front of the coded data of the camera image whilethe coded data of I-frame is located at a position upstream by It3 bytesfrom the aforementioned position. If the decoder needs to start displayfrom time t3, the coded data of the I-frame located at the (Bt3-It3)-thbyte from the front is decoded first. Then, P-frames and B-frames aresuccessively decoded while counting the number of bytes of the decodeddata until the count reaches It3 bytes. When the display is started atthis point of time, the image data from the designated time t3 isdisplayed.

Next, other accessing methods will be described.

-   (A) Coded data is packetized and the header information of each    packet has information that indicates whether the packet contains    the front of an I-frame. In FIG. 13(b), the designated time and the    number of bytes to the packet corresponding to the designated time    are written. When the decoder had made access to the packet    corresponding to the designated time t3, it is checked as to whether    the packet contains the front of an I-frame. The decoder starts    decoding and display from a packet that contains an I-frame (all the    packets before that are discarded as unnecessary).-   (B) In (A), instead of indicating the number of bytes to the packet,    only the packet number is written in FIG. 13(b).

The length of the packets in a piece of coded data is assumed to befixed and the number of bytes of one packet is written in the headerinformation of the coded data. The decoder calculates the number ofbytes to the packet corresponding to the designated time based on thepacket number and the number of bytes of one packet (thereafter thesteps are the same as (A)).

Next, other storage forms will be described.

-   (C) In FIG. 12, the management information and the coded information    are joined and stored. However, the management information may be    separated and stored as a different file.-   (D) Of the management information, the information for access to    designated times may be included in the header information of the    coded data of each camera image, instead of being included in the    management information. In this case, the third column in FIG. 13(a)    (the pointer to the information for access to designated times    within each camera image) is not necessary.-   (E) The management information, the coded data of individual camera    images may be all separated into different files. In this case, the    number of bytes from the front in the second column in FIG. 13(a) is    replaced by the filename of the coded data of each camera image, for    example. Access to each camera image is made based on the filename.

INDUSTRIAL APPLICABILITY

The image data distribution system, its image data transmittingapparatus and image data receiving apparatus according to the presentinvention does not need large amounts of memory and a high-performanceCPU on the client side, and can be applied to mobile terminals whichenable observations of a stereoscopic image from arbitrary viewpoints.

1. An image data distribution system comprising: a request informationreceiving means for receiving client's request information transmittedby way of a network; a request information analyzing means for analyzingthe request information received by the request information receivingmeans; a multiple viewpoint image supply means which selects necessaryimage data from coded and stored multiple viewpoint image data of imagestaken by a plurality of cameras, based on viewpoint information from therequest information analyzed by the request information analyzing meansand decodes and supplies selected data; an image generating means which,based on image data supplied from the multiple viewpoint image supplyingmeans, generates image data of an image viewed from a predeterminedviewpoint in conformity with the request information; an imagesynthesizing means for synthesizing a plurality of images data generatedby the image generating means, based on display unit information fromthe request information; a coding means for coding image datasynthesized by the image synthesizing means; a transmitting means fortransmitting coded image data by the coding means to the network; areceiving means for receiving the coded image data via the network; adecoding means for decoding the coded image data received by thereceiving means; an image processing means for processing decoded imagedata by the decoding means so as to be display able on a display means;the display means for displaying image data processed by the imageprocessing means; a request information input means for allowing inputof the client's request information; and a request informationtransmitting means for transmitting the request information to thenetwork.
 2. An image data transmitting apparatus comprising: a requestinformation receiving means for receiving client's request informationtransmitted by way of a network; a request information analyzing meansfor analyzing the request information received by the requestinformation receiving means; a multiple viewpoint image supply meanswhich selects necessary image data from coded and storedmultipleviewpoint image data of images taken by a plurality of cameras, based onview point information from the request information analyzed by therequest information analyzing means and decodes and supplies selecteddata; an image generating means which, based on image data supplied fromthe multiple viewpoint image supplying means, generates image data of animage viewed from a predetermined viewpoint in conformity with therequest information; an image synthesizing means for synthesizing aplurality of images data generated by the image generating means, basedon display unit information from the request information; a coding meansfor coding image data synthesized by the image synthesizing means; and atransmitting means for transmitting coded image data by the coding meansto the network.
 3. (canceled)
 4. The image data distributing systemaccording to claim 1, further comprising a management information addingmeans for adding management information for enabling access to the imagedata of individual viewpoints and random access, to the multipleviewpoint image data.
 5. The image data transmitting apparatus accordingto claim 2, further comprising a management information adding means foradding management information for enabling access to the image data ofindividual viewpoints and random access, to the multiple viewpoint imagedata.
 6. An image data distribution system comprising: a requestinformation receiving means for receiving client's request informationtransmitted by way of a network; a request information analyzing meansfor analyzing the request information received by the requestinformation receiving means; a multiple viewpoint image supply means forsupplying multiple viewpoint image data; an image generating meanswhich, based on viewpoint information from the request informationanalyzed by the request information analyzing means, receives input ofnecessary image data from the multiple viewpoint image supply means andgenerates image data of an image viewed from a predetermined viewpointin conformity with the request information; an image synthesizing meansfor synthesizing a plurality of images data generated by the imagegenerating means, based on display unit information from the requestinformation; a coding means for coding image data synthesized by theimage synthesizing means; a transmitting means for transmitting codedimage data by the coding means to the network; a receiving means forreceiving the coded image data via the network; a decoding means fordecoding the coded image data received by the receiving means; an imageprocessing means for processing decoded image data by the decoding meansso as to be display able on a display means; the display means fordisplaying image data processed by the image processing means; a requestinformation input means for allowing input of the client's requestinformation; a request information transmitting means for transmittingthe request information to the network; and a judgement means forjudging whether the received image data is of two-dimensional image dataor stereoscopic image data.
 7. An image data receiving apparatuscomprising: a receiving means for receiving coded image data by way of anetwork; a decoding means for decoding the coded image data received bythe receiving means; an image processing means for processing decodedimage data by the decoding means so as to be display able on a displaymeans; the display means for displaying image data processed by theimage processing means; a request information input means for allowinginput of request information of a client; a request informationtransmitting means for transmitting the request information to thenetwork; and a judgement means for judging whether the received imagedata is of two-dimensional image data or stereoscopic image data.
 8. Animage data distribution system comprising: a request informationreceiving means for receiving client's request information transmittedby way of a network; a request information analyzing means for analyzingthe request information received by the request information receivingmeans; a multiple viewpoint image supply means for supplying multipleviewpoint image data; an image generating means which, based onviewpoint information from the request information analyzed by therequest information analyzing means, receives input of necessary imagedata from the multiple viewpoint image supply means and generates imagedata of an image viewed from a predetermined viewpoint in conformitywith the request information; an image synthesizing means forsynthesizing a plurality of images data generated by the imagegenerating means, based on display unit information from the requestinformation; a coding means for coding image data synthesized by theimage synthesizing means; a transmitting means for transmitting codedimage data by the coding means to the network; a receiving means forreceiving the coded image data via the network; a decoding means fordecoding the coded image data received by the receiving means; an imageprocessing means for processing decoded image data by the decoding meansso as to be display able on a display means; the display means fordisplaying image data processed by the image processing means; a requestinformation input means for allowing input of the client's requestinformation; a request information transmitting means for transmittingthe request information to the network; and an identificationinformation adding means for adding to the image data to be transmitteda piece of information that indicates whether the image data is oftwo-dimensional image data or stereoscopic image data.
 9. An image datatransmitting apparatus comprising: a request information receiving meansfor receiving client's request information transmitted by way of anetwork; a request information analyzing means for analyzing the requestinformation received by the request information receiving means; amultiple viewpoint image supply means for supplying multiple viewpointimage data; an image generating means which, based on viewpointinformation from the request information analyzed by the requestinformation analyzing means, receives input of necessary image data fromthe multiple viewpoint image supply means and generates image data of animage viewed from a predetermined viewpoint in conformity with therequest information; an image synthesizing means for synthesizing aplurality of images data generated by the image generating means, basedon display unit information from the request information; a coding meansfor coding image data synthesized by the image synthesizing means; atransmitting means for transmitting coded image data by the coding meansto the network; and an identification information adding means foradding to the image data to be transmitted a piece of information thatindicates whether the image data is of two-dimensional image data orstereoscopic image data.
 10. An image data receiving apparatuscomprising: a receiving means for receiving coded image data by way of anetwork; a decoding means for decoding the coded image data received bythe receiving means; an image processing means for processing decodedimage data by the decoding means so as to be display able on a displaymeans; the display means for displaying image data processed by theimage processing means; a request information input means for allowinginput of request information of a client; a request informationtransmitting means for transmitting the request information to thenetwork; and an identification information adding means for adding tothe received image data a piece of information that indicates whetherthe image data is of two-dimensional image data or stereoscopic imagedata.